##################################################################

Edit to provide follow up to the answer. I modified my code as follows:

def map_func(s): flat_comments = praw.helpers.flatten_tree(s.comments) g = redditThreadDetailed(s.title, s.ups, s.created_utc, s.id, s.subreddit, flat_comments[:COMMENT_LIMIT], r) g.get_comments_info() dictD = g.dict del dictD["praw_object"] dictD["users"] = [user.dict for user in dictD["users"]] map(lambda d: d.pop("praw_object"), dictD["users"] ) user_array = dictD["users"] dictD["comments"] = list(dictD["comments"]) for user in dictD["users"]: user["threads"] = [thread.dict for thread in user["threads"]] user["comments"] = [comment.dict for comment in user["comments"]] return dictD

submissions = get_submissions(1)
with futures.ProcessPoolExecutor(max_workers=4) as executor: 
#or using futures.ThreadPoolExecutor
    for s in executor.map(map_func, submissions):
        collection_front.update({"time_recorded":time_recorded}, {'$push':{"thread_list":s}}, upsert=True)

And then I was able to see an improvement down from 51 seconds under test conditions for no threading to 44 seconds with threading to 6 seconds with multiple processes! However the speeding up of the threading and multiple processes did not depend on the max_workers parameter, which makes me think the PRAW API or Reddit limits this in some way to what works out to 2 on my machine. For future reference, I'm working on Ubuntu AWS EC2 Ubuntu instance in case this is machine specific. Multiple processes are definitely much better than multiple threads.

##################################################################

Edit to provide follow up to the answer. I modified my code as follows:

def map_func(s): flat_comments = praw.helpers.flatten_tree(s.comments) g = redditThreadDetailed(s.title, s.ups, s.created_utc, s.id, s.subreddit, flat_comments[:COMMENT_LIMIT], r) g.get_comments_info() dictD = g.dict del dictD["praw_object"] dictD["users"] = [user.dict for user in dictD["users"]] map(lambda d: d.pop("praw_object"), dictD["users"] ) user_array = dictD["users"] dictD["comments"] = list(dictD["comments"]) for user in dictD["users"]: user["threads"] = [thread.dict for thread in user["threads"]] user["comments"] = [comment.dict for comment in user["comments"]] return dictD

submissions = get_submissions(1)
with futures.ProcessPoolExecutor(max_workers=4) as executor: 
#or using futures.ThreadPoolExecutor
    for s in executor.map(map_func, submissions):
        collection_front.update({"time_recorded":time_recorded}, {'$push':{"thread_list":s}}, upsert=True)

And then I was able to see an improvement down from 51 seconds under test conditions for no threading to 44 seconds with threading to 6 seconds with multiple processes! However the speeding up of the threading and multiple processes did not depend on the max_workers parameter, which makes me think the PRAW API or Reddit limits this in some way to what works out to 2 on my machine. For future reference, I'm working on Ubuntu AWS EC2 Ubuntu instance in case this is machine specific. Multiple processes are definitely much better than multiple threads.

I would like to improve my Reddit-scraping code to make it faster, but I don't know how. I am using deque instead of list to improve append performance. Otherwise my code calls the PRAW api and type checks one class attribute (I don't know why this is necessary but otherwise I get type errors). Is there a faster way to do this? These functions are surprisingly slow, and performance time is also highly variable in addition to having a high mean execution time.

For a given user, I'm trying to retrieve their most recent comments and posts. I've pasted the code at the bottom of this post, but I have two problems. First, the code seems to run awfully slow considering how little information I'm retrieving. Second, the variance in execution time from one function call to the next is surprisingly high. You'll see I have two sleep commands for 1 second each, so the minimum time for get_user_comments_and_posts() to execute is 2 seconds. Sometimes I see this, but sometimes I see 14 seconds! And when I print out the output for a given function call, there doesn't seem to be anything special about the output for a short vs. long execution time.

Here's the code:

this comment has 0 votes and belongs in the todayilearned subreddit where it was made at 1434032968
this comment has 1 votes and belongs in the politics subreddit where it was made at 1434026834
'get_comments'  2.43 sec
'get_s_praw_call'  0.00 sec
'__init__'  0.00 sec
'__init__'  0.00 sec
this has title: Rupert Murdoch stepping down from 21st Century Fox and came from subreddit: news
this has title: Top athletics coach Alberto Salazar faces doping claims and came from subreddit: news
'get_submissions'  0.62 sec
'get_user_comments_and_posts'  5.05 sec
'get_c_praw_call'  0.09 sec
'__init__'  0.00 sec
'__init__'  0.00 sec
this comment has 1 votes and belongs in the todayilearned subreddit where it was made at 1434044423
this comment has 1 votes and belongs in the todayilearned subreddit where it was made at 1434044125
'get_comments'  2.17 sec
'get_s_praw_call'  0.00 sec
'__init__'  0.00 sec
'__init__'  0.00 sec
this has title: [SunTuThursday Forum] Halodrol - DAA - Sub-Q Injections - Squid vs. Octopus vs. Cuttlefish: Which is the ultimate cephalod? - Ask Anything! and came from subreddit: steroids
this has title: [Off-Topic Discussion] "Everyone has an opinion, and the guy screaming for censorship may be the next guy to have his ideas cut off." - Richard King and came from subreddit: steroids
'get_submissions'  0.86 sec
'get_user_comments_and_posts'  5.03 sec
'get_c_praw_call'  0.05 sec
'__init__'  0.00 sec
'__init__'  0.00 sec
this comment has 0 votes and belongs in the PaoMustResign subreddit where it was made at 1434040381
this comment has 1 votes and belongs in the Art subreddit where it was made at 1434038905
'get_comments'  2.08 sec
'get_s_praw_call'  0.00 sec
'__init__'  0.00 sec
'__init__'  0.00 sec
this has title: How much lower of an iq is the average FPH poster? and came from subreddit: circlebroke
this has title: When will GallowBoob reach Bill Brasky status? and came from subreddit: circlebroke
'get_submissions'  0.99 sec
'get_user_comments_and_posts'  5.07 sec

So it is often taking upwards of 3 seconds to process the API calls. Is this normal or am I making some really stupid blunder?

Thanks for any and all suggestions and critiques. I would like to write better Python so all feedback would be appreciated.

I would like to improve my Reddit-scraping code to make it faster, but I don't know how. I am using deque instead of list to improve append performance. Otherwise my code calls the PRAW API and type checks one class attribute (I don't know why this is necessary but otherwise I get type errors). Is there a faster way to do this? These functions are surprisingly slow, and performance time is also highly variable in addition to having a high mean execution time.

For a given user, I'm trying to retrieve their most recent comments and posts. I've pasted the code at the bottom of this post, but I have two problems. First, the code seems to run awfully slow considering how little information I'm retrieving. Second, the variance in execution time from one function call to the next is surprisingly high. You'll see I have two sleep commands for 1 second each, so the minimum time for get_user_comments_and_posts() to execute is 2 seconds. Sometimes I see this, but sometimes I see 14 seconds! And when I print out the output for a given function call, there doesn't seem to be anything special about the output for a short vs. long execution time.

this comment has 0 votes and belongs in the todayilearned subreddit where it was made at 1434032968
this comment has 1 votes and belongs in the politics subreddit where it was made at 1434026834
'get_comments'  2.43 sec
'get_s_praw_call'  0.00 sec
'__init__'  0.00 sec
'__init__'  0.00 sec
this has title: Rupert Murdoch stepping down from 21st Century Fox and came from subreddit: news
this has title: Top athletics coach Alberto Salazar faces doping claims and came from subreddit: news
'get_submissions'  0.62 sec
'get_user_comments_and_posts'  5.05 sec
'get_c_praw_call'  0.09 sec
'__init__'  0.00 sec
'__init__'  0.00 sec
this comment has 1 votes and belongs in the todayilearned subreddit where it was made at 1434044423
this comment has 1 votes and belongs in the todayilearned subreddit where it was made at 1434044125
'get_comments'  2.17 sec
'get_s_praw_call'  0.00 sec
'__init__'  0.00 sec
'__init__'  0.00 sec
this has title: [SunTuThursday Forum] Halodrol - DAA - Sub-Q Injections - Squid vs. Octopus vs. Cuttlefish: Which is the ultimate
cephalod? - Ask Anything! and came from subreddit: steroids
this has title: [Off-Topic Discussion] "Everyone has an opinion, and the guy screaming for censorship may be the next guy to have his
ideas cut off." - Richard King and came from subreddit: steroids
'get_submissions'  0.86 sec
'get_user_comments_and_posts'  5.03 sec
'get_c_praw_call'  0.05 sec
'__init__'  0.00 sec
'__init__'  0.00 sec
this comment has 0 votes and belongs in the PaoMustResign subreddit where it was made at 1434040381
this comment has 1 votes and belongs in the Art subreddit where it was made at 1434038905
'get_comments'  2.08 sec
'get_s_praw_call'  0.00 sec
'__init__'  0.00 sec
'__init__'  0.00 sec
this has title: How much lower of an iq is the average FPH poster? and came from subreddit: circlebroke
this has title: When will GallowBoob reach Bill Brasky status? and came from subreddit: circlebroke
'get_submissions'  0.99 sec
'get_user_comments_and_posts'  5.07 sec

So it is often taking upwards of 3 seconds to process the API calls. Is this normal or am I making some really stupid blunder?