I write an hobbyist application that collects soccer-data from different sources around the www and aggregates it for a graphical representation and sends it out as a tweet. Now I want to fully automate this process in a more or less robust way (more or less because 1. this is a hobby and 2. depends on scraped 3rd party websites).
I have problems to wrap my head around how this full automation can be organized. The following tasks need to be done:
- check once daily if and when there are soccer matches I am interested in
- start collecting data of these matches from all sources as soon as match is over
- retry collecting data every x minutes if it wasn't available yet (the sources have very different speed in releasing the data and it may take 24hrs. sometimes)
- as soon as all sources have been collected successfully launch aggregation and tweet.
I don't know how to implement the connection between 3. and 4. in a good way. I think I need some wrapper which is called by the scheduler as soon as a match is over, but this doesn't seem to be a robust way:
class MatchCollector: def __init__(self,teams,sources=settings.sources): self.teams=teams self.sources = sources def execute(self): finished = False counter = 0 while not finished: finished = True for source in self.sources: s = SourceCollector(source,self.teams) successful = s.execute() if not successful: finished = False time.sleep(600) counter += counter if counter == 10000 # arbitrary chosen, have to make calc, when would be a good point to give up raise IterationException('Tried for x hours without success. Somethings broken') d = DataAggregator(sources) d.execute()
Am I missing something problematic, by doing this?
I arrange daily scheduling (1., 2.) via cron and python-crontab in a scheduling script. It is started once daily and reads the daily matches and writes again into cron the script to handle the single matches with arguments passed via sys.argv.