Aim: To improve the speed of the following code. Current timing is about 80~ hours :0

Purpose: The code imports a dataset which contains 1.9 million rows and two columns. One of these columns contain text posts of var length. I then loop through each of these rows and query the post against an imported function that returns a specific counter of variable length. The counter tells me about the presence of certain words in the text. On average the func takes less than 1 ms to return this counter. (Timer for the "Func" inserted at the end to prove this)

Overheads: The code i'm looking to improve is the loop. I accept a certain level of overhead with the "func" which can't be improved at this minute. I have considered looking at Spark or Dask to parallelize the loop and speed up the process. Suggestions are welcome

#Import data
import pandas as pd 
from func import func 
data = pd.read_csv('Dataset.csv')

>> 1900000

>> Index(['type', 'body'], dtype='object')

#Create new DF
data2 = pd.Dataframe()

for post in data['post']:
    post = str(post)
    scores = func.countWords(posts)
    data2 = data2.append(scores,ignore_index=True)

>> Counter({0: 306,
     1: 185,
     2: 61,
     45: 31,
     87: 23,
     92: 5,
     94: 3,
     102: 30,})

 import time
 start = time.time()
 score = func.countWords("Slow down Sir, you're going to give yourself skin faliure!")
 end = time.time()
 print(end - start)
 >> 0.0019948482513427734
  • 1
    \$\begingroup\$ Any appreciable speedup here will come from optimizing countWords - I"d advise that you post that function for review. \$\endgroup\$ – Zach Jul 31 '18 at 16:56

Several errors:

I then loop through each of these loops

I take it you mean "I then loop through each of these columns".

for post in data['post]:

missing end quote mark

scores = Func.countWords(posts)

You imported func (lowercase) and now you're calling Func (uppercase)

data2 = data2.append(scores,ignore_index=True)

append should take a row-type object. If the function returns a numeric, then you shouldn't be appending it. Instead you can do:

def post_to_count(post):
      return func.countWord(str(post))

scores = data['post'].apply(post_to_count)
  • \$\begingroup\$ I've fixed those spelling and syntaxs errors. Your .apply works fine, but 1) it returns a combined dataframe, 2) does it increase speed up? \$\endgroup\$ – F.D Jul 31 '18 at 16:16
  • \$\begingroup\$ What about scores = data.post.str.apply(func.countWord)? \$\endgroup\$ – Mathias Ettinger Jul 31 '18 at 20:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.