I wrote a script for getting the stats of codereview.SE from the front page into a file. Github link if someone prefers reading there. Here is
data_file.txt's current contents
23-08-2013 dd-mm-yyyy, questions, answers, %answered, users, visitors/day 22-08-2013,9079,15335,88,26119,7407, 23-08-2013,9094,15354,88,26167,7585,
The first sentence contains the last date at which the data is written into the file. It is a check to make sure that I don't write the data for the same date twice in the file. Also note that there is a newline at the end which isn't being shown above.
What I plan to do is to collect the data for many data and then use
matplotlib to get graphs for finding out about the growth of this website. The part about plotting is currently under progress so I am not putting that here.
The python script that I am using is below. I would like a general review of this script and any suggestions about its design in particular.
#! python3 import urllib.request import datetime FILE_NAME = 'data_file.txt' CURRENT_URL = 'http://codereview.stackexchange.com/' def today_date(): return datetime.date.today().strftime('%d-%m-%Y') def already_written(): with open(FILE_NAME, 'a+') as f: f.seek(0) first_line = f.readline() if today_date() == first_line[:-1]: return True return False def parse(line): """This separates the stat-name and associated number""" temp = [0, ''] braces = False for c in line: if c == '<': braces = True elif c == '>': braces = False elif braces is True or c in [' ', ',', '%']: continue elif c.isdigit(): temp *= 10 temp += int(c) else: temp += c return temp def write_stats(): '''This writes the stats into the file''' with open(FILE_NAME, 'r') as f: data = f.readlines() with open(FILE_NAME, 'w') as f: f.write(today_date() + '\n') f.writelines(data[1:]) url_handle = urllib.request.urlopen(CURRENT_URL) write_this = today_date() + ',' for line in url_handle: temp_line = str(line)[2:-5] if 'stats-value' in temp_line and 'label' in temp_line: temp = parse(temp_line) write_this += str(temp) + ',' else: write_this += '\n' f.write(write_this) def main(): if not already_written(): write_stats() if __name__ == "__main__": main()
I think this line might need some explanation.
if 'stats-value' in temp_line and 'label' in temp_line:
I choose this for finding out the lines in the source code of the html of codereview.SE's front page. Only the 5 lines containing the stats have these 2 in it. So in front they tell me which line to parse.