As far as I can tell, there is no network access happening at this stage in the code. I am accessing Reddit's API via the PRAW module. However, it is crawling, and I think it should be faster, considering I am hardly doing any work (unless Python heavily penalizes object access?).
This is the entire short script:
import sys, os, pprint, praw
class Scanner(object):
''' A scanner object. '''
def __init__(self):
self.user_agent = 'debian.blah8899889.agent'
self.r = praw.Reddit(user_agent=self.user_agent)
self.nsfw = ('funny', 'nsfw')
self.nsfw_posters = set()
self.content = []
def getSub(self, subreddit):
''' Accepts a subreddit. Adds subreddit posts object to self.content'''
url = 'http://www.reddit.com/r/{sub}/'.format(sub=subreddit)
print 'Scanning:', subreddit
subreddit_posts = self.r.get_content(url, limit=5)
self.addContent(subreddit_posts)
def addContent(self, subreddit):
print 'Adding subreddit posts to content.'
self.content.append(subreddit)
def addNSFWPoster(self, post):
print 'Parsing author and adding to posters.'
self.nsfw_posters.add(str(post.author))
def scanNSFW(self):
''' Scans all NSFW subreddits. Makes list of posters. '''
# Get content from all nsfw subreddits
print 'Executing map function.'
map(self.getSub, self.nsfw)
# Scan content and get authors
print 'Executing list comprehension.'
[self.addNSFWPoster(post) for sub in self.content for post in sub]
def main():
scan = Scanner()
scan.scanNSFW()
main()
All of the network access should happen at map(self.getSub, self.nsfw)
. This actually runs quite fast considering I am rate limited by Reddit's servers.
I cannot work out why the list comprehension is so slow. All it should be doing is iterating through some objects and extracting a simple attribute: it should merely get str(post.author)
.
[self.addNSFWPoster(post) for sub in self.content for post in sub]
Can someone help me understand why it seems so slow to execute?
I am new to the OOP approach, and I can only assume I have made a big mistake somewhere. It is slower than the network access portion of the code!