3
\$\begingroup\$

As far as I can tell, there is no network access happening at this stage in the code. I am accessing Reddit's API via the PRAW module. However, it is crawling, and I think it should be faster, considering I am hardly doing any work (unless Python heavily penalizes object access?).

This is the entire short script:

import sys, os, pprint, praw

class Scanner(object):
    ''' A scanner object. '''
    def __init__(self):
        self.user_agent = 'debian.blah8899889.agent'
        self.r = praw.Reddit(user_agent=self.user_agent)
        self.nsfw = ('funny', 'nsfw')
        self.nsfw_posters = set()
        self.content = []

    def getSub(self, subreddit):
        ''' Accepts a subreddit. Adds subreddit posts object to self.content'''
        url = 'http://www.reddit.com/r/{sub}/'.format(sub=subreddit)
        print 'Scanning:', subreddit
        subreddit_posts = self.r.get_content(url, limit=5)
        self.addContent(subreddit_posts)

    def addContent(self, subreddit):
        print 'Adding subreddit posts to content.'
        self.content.append(subreddit)

    def addNSFWPoster(self, post):
        print 'Parsing author and adding to posters.'
        self.nsfw_posters.add(str(post.author))

    def scanNSFW(self):
        ''' Scans all NSFW subreddits. Makes list of posters. '''
#       Get content from all nsfw subreddits
        print 'Executing map function.'
        map(self.getSub, self.nsfw)
#       Scan content and get authors
        print 'Executing list comprehension.'
        [self.addNSFWPoster(post) for sub in self.content for post in sub]

def main():
    scan = Scanner()
    scan.scanNSFW()

main()

All of the network access should happen at map(self.getSub, self.nsfw). This actually runs quite fast considering I am rate limited by Reddit's servers.

I cannot work out why the list comprehension is so slow. All it should be doing is iterating through some objects and extracting a simple attribute: it should merely get str(post.author).

 [self.addNSFWPoster(post) for sub in self.content for post in sub]

Can someone help me understand why it seems so slow to execute?

I am new to the OOP approach, and I can only assume I have made a big mistake somewhere. It is slower than the network access portion of the code!

\$\endgroup\$
0

3 Answers 3

5
\$\begingroup\$

Well for starters with

[self.addNSFWPoster(post) for sub in self.content for post in sub]

you are constructing a list of

[None, None, ... , None]

with a length equal to

len(self.content) * len(sub)

that you are immediately throwing away because you aren't assigning it to a variable. In other words you should only use a list comprehension for building a list, not for calling a method that is doing work elsewhere.

An alternative implementation is as you might expect a nested for loop:

for sub in self.content:
    for post in sub:
        self.addNSFWPoster(post)

To speed this up further you could reduce your function call overhead with the following:

for sub in self.content:
    for post in sub:
        print 'Parsing author and adding to posters.'
        self.nsfw_posters.add(str(post.author))
\$\endgroup\$
4
\$\begingroup\$

Don't call main() directly from the program like this. Place this call in an if block like this:

if __name__ == "__main__":
    main()

This helps prevent problems created when you use multiple files, and is explained here.

\$\endgroup\$
1
\$\begingroup\$

All of the network access should happen at map(self.getSub, self.nsfw). This actually runs quite fast considering I am rate limited by Reddit's servers.

I cannot work out why the list comprehension is so slow. All it should be doing is iterating through some objects and extracting a simple attribute: it should merely get str(post.author).

See the PRAW docs on Lazy Objects and get_content. Data is fetched from Reddit when it is needed in the list comprehension. for post in sub iterates over a generator returned from get_content.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.