I wrote a script for getting the stats of codereview.SE from the front page into a file. Github link if someone prefers reading there. Here is data_file.txt
's current contents
23-08-2013
dd-mm-yyyy, questions, answers, %answered, users, visitors/day
22-08-2013,9079,15335,88,26119,7407,
23-08-2013,9094,15354,88,26167,7585,
The first sentence contains the last date at which the data is written into the file. It is a check to make sure that I don't write the data for the same date twice in the file. Also note that there is a newline at the end which isn't being shown above.
What I plan to do is to collect the data for many data and then use matplotlib
to get graphs for finding out about the growth of this website. The part about plotting is currently under progress so I am not putting that here.
The python script that I am using is below. I would like a general review of this script and any suggestions about its design in particular.
#! python3
import urllib.request
import datetime
FILE_NAME = 'data_file.txt'
CURRENT_URL = 'http://codereview.stackexchange.com/'
def today_date():
return datetime.date.today().strftime('%d-%m-%Y')
def already_written():
with open(FILE_NAME, 'a+') as f:
f.seek(0)
first_line = f.readline()
if today_date() == first_line[:-1]:
return True
return False
def parse(line):
"""This separates the stat-name and associated number"""
temp = [0, '']
braces = False
for c in line:
if c == '<':
braces = True
elif c == '>':
braces = False
elif braces is True or c in [' ', ',', '%']:
continue
elif c.isdigit():
temp[0] *= 10
temp[0] += int(c)
else:
temp[1] += c
return temp
def write_stats():
'''This writes the stats into the file'''
with open(FILE_NAME, 'r') as f:
data = f.readlines()
with open(FILE_NAME, 'w') as f:
f.write(today_date() + '\n')
f.writelines(data[1:])
url_handle = urllib.request.urlopen(CURRENT_URL)
write_this = today_date() + ','
for line in url_handle:
temp_line = str(line)[2:-5]
if 'stats-value' in temp_line and 'label' in temp_line:
temp = parse(temp_line)
write_this += str(temp[0]) + ','
else:
write_this += '\n'
f.write(write_this)
def main():
if not already_written():
write_stats()
if __name__ == "__main__":
main()
EDIT:
I think this line might need some explanation.
if 'stats-value' in temp_line and 'label' in temp_line:
I choose this for finding out the lines in the source code of the html of codereview.SE's front page. Only the 5 lines containing the stats have these 2 in it. So in front they tell me which line to parse.