Review Python script for getting the stats of codereview.SE into a file for analysis

Question

I wrote a script for getting the stats of codereview.SE from the front page into a file. Github link if someone prefers reading there. Here is data_file.txt's current contents

23-08-2013
dd-mm-yyyy, questions, answers, %answered, users, visitors/day

22-08-2013,9079,15335,88,26119,7407,
23-08-2013,9094,15354,88,26167,7585,

The first sentence contains the last date at which the data is written into the file. It is a check to make sure that I don't write the data for the same date twice in the file. Also note that there is a newline at the end which isn't being shown above.

What I plan to do is to collect the data for many data and then use matplotlib to get graphs for finding out about the growth of this website. The part about plotting is currently under progress so I am not putting that here.

The python script that I am using is below. I would like a general review of this script and any suggestions about its design in particular.

#! python3
import urllib.request
import datetime

FILE_NAME = 'data_file.txt'
CURRENT_URL = 'http://codereview.stackexchange.com/'

def today_date():
    return datetime.date.today().strftime('%d-%m-%Y')


def already_written():
    with open(FILE_NAME, 'a+') as f:
        f.seek(0)
        first_line = f.readline()
        if today_date() == first_line[:-1]:
            return True
        return False


def parse(line):
    """This separates the stat-name and associated number"""
    temp = [0, '']
    braces = False
    for c in line:
        if c == '<':
            braces = True
        elif c == '>':
            braces = False
        elif braces is True or c in [' ', ',', '%']:
            continue
        elif c.isdigit():
            temp[0] *= 10
            temp[0] += int(c)
        else:
            temp[1] += c

    return temp


def write_stats():
    '''This writes the stats into the file'''
    with open(FILE_NAME, 'r') as f:
        data = f.readlines()

    with open(FILE_NAME, 'w') as f:
        f.write(today_date() + '\n')
        f.writelines(data[1:])

        url_handle = urllib.request.urlopen(CURRENT_URL)
        write_this = today_date() + ','
        for line in url_handle:
            temp_line = str(line)[2:-5]
            if 'stats-value' in temp_line and 'label' in temp_line:
                temp = parse(temp_line)
                write_this += str(temp[0]) + ','
        else:
            write_this += '\n'
            f.write(write_this)

def main():
    if not already_written():
        write_stats()

if __name__ == "__main__":
    main()

EDIT:

I think this line might need some explanation.

if 'stats-value' in temp_line and 'label' in temp_line:

I choose this for finding out the lines in the source code of the html of codereview.SE's front page. Only the 5 lines containing the stats have these 2 in it. So in front they tell me which line to parse.

Answer 1

Firstly, you don't need to store the last date at the beginning of the file since it's already at the end.

Secondly, you shouldn't compute today's date twice.

Thirdly, you should use Stack Exchange's API instead of trying to extract information from Code Review's home page. It has everything except visitors/day.

Fourthly, the date format %Y-%m-%d is better than %d-%m-%Y because it allows string comparison of dates.

So, here's what I came up with:

#!/usr/bin/env python3

import datetime
import gzip
import json
from urllib.request import Request, urlopen

FILE_NAME = 'data_file.txt'
API_URL = 'http://api.stackexchange.com/2.1/info?site=codereview'

def last(iterable, default):
    r = default
    for v in iterable:
        r = v
    return r

def main():
    today = datetime.date.today().strftime('%Y-%m-%d')
    with open(FILE_NAME, 'a+') as f:
        f.seek(0)
        last_date = last(f, '').split(',')[0]
        if last_date == today:
            return  # we already have the data for today
        elif last_date > today:
            raise Exception('last date is in the future')

        req = Request(API_URL, headers={'Accept-Encoding':'gzip'})
        r = gzip.decompress(urlopen(req).read()).decode('utf8')
        d = json.loads(r)['items'][0]
        print(today,
            d['total_questions'],
            d['total_answers'],
            d['total_unanswered'],
            d['total_users'],
            sep=',', file=f)

if __name__ == "__main__":
    main()

The gzip stuff is to work around a bug in Stack Exchange's API server: it always compresses the response even if you don't ask, which is a protocol violation. Using the requests library might make it easier.

The way I get the last line of the file isn't the most efficient but it should be good enough.