Aim: To improve the speed of the following code. Current timing is about 80~ hours :0

Purpose: The code imports a dataset which contains 1.9 million rows and two columns. One of these columns contain text posts of var length. I then loop through each of these rows and query the post against an imported function that returns a specific counter of variable length. The counter tells me about the presence of certain words in the text. On average the func takes less than 1 ms to return this counter. (Timer for the "Func" inserted at the end to prove this)

Overheads: The code i'm looking to improve is the loop. I accept a certain level of overhead with the "func" which can't be improved at this minute. I have considered looking at Spark or Dask to parallelize the loop and speed up the process. Suggestions are welcome

#Import data
import pandas as pd 
from func import func 
data = pd.read_csv('Dataset.csv')

>> 1900000

>> Index(['type', 'body'], dtype='object')

#Create new DF
data2 = pd.Dataframe()

for post in data['post']:
    post = str(post)
    scores = func.countWords(posts)
    data2 = data2.append(scores,ignore_index=True)

>> Counter({0: 306,
     1: 185,
     2: 61,
     45: 31,
     87: 23,
     92: 5,
     94: 3,
     102: 30,})

 import time
 start = time.time()
 score = func.countWords("Slow down Sir, you're going to give yourself skin faliure!")
 end = time.time()
 print(end - start)
 >> 0.0019948482513427734
  • 4
    \$\begingroup\$ Any appreciable speedup here will come from optimizing countWords - I"d advise that you post that function for review. \$\endgroup\$ – Zach Jul 31 '18 at 16:56
  • 1
    \$\begingroup\$ This question is being discussed on meta \$\endgroup\$ – IEatBagels Aug 14 '19 at 13:04

Several errors:

I then loop through each of these loops

I take it you mean "I then loop through each of these columns".

for post in data['post]:

missing end quote mark

scores = Func.countWords(posts)

You imported func (lowercase) and now you're calling Func (uppercase)

data2 = data2.append(scores,ignore_index=True)

append should take a row-type object. If the function returns a numeric, then you shouldn't be appending it. Instead you can do:

def post_to_count(post):
      return func.countWord(str(post))

scores = data['post'].apply(post_to_count)
| improve this answer | |
  • \$\begingroup\$ I've fixed those spelling and syntaxs errors. Your .apply works fine, but 1) it returns a combined dataframe, 2) does it increase speed up? \$\endgroup\$ – F.D Jul 31 '18 at 16:16
  • \$\begingroup\$ What about scores = data.post.str.apply(func.countWord)? \$\endgroup\$ – 301_Moved_Permanently Jul 31 '18 at 20:05
  • 3
    \$\begingroup\$ Hey there, you've answered the question in a state when it was off-topic for the site. In accordance with existing site policy the edit to the question that makes your answer basically moot is acceptable. Considering that your answer in it's current state doesn't really contain a lot of useful suggestions for the latest revision of the post, you may want to edit your answer to avoid downvotes (or the answer being deleted). For more information, see this meta discussion. Thanks! \$\endgroup\$ – Vogel612 Aug 14 '19 at 13:33

Not the answer you're looking for? Browse other questions tagged or ask your own question.