# Review Python script for getting the stats of codereview.SE into a file for analysis

I wrote a script for getting the stats of codereview.SE from the front page into a file. Github link if someone prefers reading there. Here is data_file.txt's current contents

23-08-2013

22-08-2013,9079,15335,88,26119,7407,
23-08-2013,9094,15354,88,26167,7585,


The first sentence contains the last date at which the data is written into the file. It is a check to make sure that I don't write the data for the same date twice in the file. Also note that there is a newline at the end which isn't being shown above.

What I plan to do is to collect the data for many data and then use matplotlib to get graphs for finding out about the growth of this website. The part about plotting is currently under progress so I am not putting that here.

The python script that I am using is below. I would like a general review of this script and any suggestions about its design in particular.

#! python3
import urllib.request
import datetime

FILE_NAME = 'data_file.txt'
CURRENT_URL = 'http://codereview.stackexchange.com/'

def today_date():
return datetime.date.today().strftime('%d-%m-%Y')

with open(FILE_NAME, 'a+') as f:
f.seek(0)
if today_date() == first_line[:-1]:
return True
return False

def parse(line):
"""This separates the stat-name and associated number"""
temp = [0, '']
braces = False
for c in line:
if c == '<':
braces = True
elif c == '>':
braces = False
elif braces is True or c in [' ', ',', '%']:
continue
elif c.isdigit():
temp[0] *= 10
temp[0] += int(c)
else:
temp[1] += c

return temp

def write_stats():
'''This writes the stats into the file'''
with open(FILE_NAME, 'r') as f:

with open(FILE_NAME, 'w') as f:
f.write(today_date() + '\n')
f.writelines(data[1:])

url_handle = urllib.request.urlopen(CURRENT_URL)
write_this = today_date() + ','
for line in url_handle:
temp_line = str(line)[2:-5]
if 'stats-value' in temp_line and 'label' in temp_line:
temp = parse(temp_line)
write_this += str(temp[0]) + ','
else:
write_this += '\n'
f.write(write_this)

def main():
write_stats()

if __name__ == "__main__":
main()


EDIT:

I think this line might need some explanation.

if 'stats-value' in temp_line and 'label' in temp_line:


I choose this for finding out the lines in the source code of the html of codereview.SE's front page. Only the 5 lines containing the stats have these 2 in it. So in front they tell me which line to parse.

• If a line needs some explanation, that explanation should be right before/beside it in a comment. :) – flornquake Aug 26 '13 at 9:05
• @flornquake Good point but I was indirectly bumping this question :) – Aseem Bansal Aug 26 '13 at 9:10

Firstly, you don't need to store the last date at the beginning of the file since it's already at the end.

Secondly, you shouldn't compute today's date twice.

Thirdly, you should use Stack Exchange's API instead of trying to extract information from Code Review's home page. It has everything except visitors/day.

Fourthly, the date format %Y-%m-%d is better than %d-%m-%Y because it allows string comparison of dates.

So, here's what I came up with:

#!/usr/bin/env python3

import datetime
import gzip
import json
from urllib.request import Request, urlopen

FILE_NAME = 'data_file.txt'
API_URL = 'http://api.stackexchange.com/2.1/info?site=codereview'

def last(iterable, default):
r = default
for v in iterable:
r = v
return r

def main():
today = datetime.date.today().strftime('%Y-%m-%d')
with open(FILE_NAME, 'a+') as f:
f.seek(0)
last_date = last(f, '').split(',')[0]
if last_date == today:
return  # we already have the data for today
elif last_date > today:
raise Exception('last date is in the future')

print(today,
d['total_questions'],